Recommendation systems can have separate information retrieval and machine-learned ranking stages. The information retrieval stage selects documents (videos, advertisements, music, text documents, etc.) from a corpus based on various signals and the machine-learned system ranks the output of the information retrieval system. For example, when a user enters a query “cat”, a contextual information retrieval system may select a set of candidate advertisements that contain the word “cat” from all available advertisements. This set of candidate advertisements can then be ranked based on a machine-learned model that has been trained to predict the likelihood of an advertisement being clicked through by a user based on various features, such as the type of user, the location of the user, the time of day at which the query was made, etc. An information retrieval tool is computationally efficient, but can only produce a rough estimate of which items are best recommended to a user. A machine-learned model can produce more accurate recommendations, but is often more computationally intensive than an information retrieval tool. Further, because the information retrieval tool is less accurate, it can exclude certain candidates from consideration using the machine-learned model that would otherwise be highly ranked.